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Abstract 

Funding  and  use  policies,  modernization  processes,  and  a  lack  of  coordination  between 
technology  developers,  system  developers,  program  offices,  and  test  organizations  all  negatively 
impact  the  ability  of  the  national  developmental  test  infrastructure’s  ability  to  support  early 
engineering  and  development  of  weapon  systems.  Accurate,  timely  world-class  test  capabilities 
are  critical,  because  the  results  affect  the  ability  of  programs  to  be  completed  successfully. 

For  programs  to  benefit,  testing  must  be  accomplished  in  a  timely  manner  and  the  test 
capability  must  be  capable  of  meeting  the  technical  needs  of  the  development  program.  The 
adequacy  of  this  capability  and  throughput  capacity  depend  heavily  upon  funding  and  use 
policies.  Funding  and  use  policies  hamper  effective  collaboration  between  developers  and 
testers  and  restrict  the  coordination  of  future  test  requirements.  These  policies  can  be  modified 
to  improve  test  facility  capability  and  capacity.  It  is  important  to  implement  these  changes  so 
that  the  Air  Force  department  can  save  time  and  money  while  improving  the  performance  of 
national  technology  and  weapon  systems  programs. 


rv 


Introduction 


A  nation’s  ability  to  field  modem  weapon  systems  depends  directly  on  the  quality  of  that 
nation’s  research,  development,  test  and  evaluation  (RDT&E)  capabilities.  The  facilities 
providing  these  capabilities  play  a  key  role  in  the  development  of  aerospace  systems.1  The  most 
critical  weapon  system  and  technology  development  programs  must  have  access  to  the  highest 
quality  development  test  capabilities.  The  Air  Force  has  spared  no  expense  developing  the  very 
best  national  scale  ground  test  capabilities  in  the  world.  However,  these  test  capabilities  are 
often  underutilized  or  unavailable  for  support  of  important  national  programs.  Funding  and  use 
policies  limit  the  utilization  of  these  test  capabilities.  For  facility  operators  to  effectively  provide 
the  needed  capacity  and  capability,  they  must  know  and  understand  the  future  workload  from 
both  technical  and  throughput  aspects.  Once  the  workload  is  understood,  funding  and  use 
processes  must  support  the  test  capability  owner’s  need  to  provide  the  required  services.  This 
paper  assesses  the  funding  and  use  processes  that  affect  the  ability  of  facility  owners  to  provide 
effective  test  capabilities  in  a  timely  manner.  Recommendations  to  improve  these  processes  are 
provided. 

Early  development  ground  test  facilities  are  critical  to  support  both  acquisition  programs 
and  technology  maturation  programs.  Developmental  testing  facilities  are  utilized  in  systems 
development  well  ahead  of  a  “Milestone  A”  program  decision  and  are  important  to  supporting 
programs  throughout  the  program  life  cycle.  These  facilities  are  a  critical  source  of  engineering 
information  for  designers,  maintainers,  operators,  and  decision  makers.  The  developmental  test 
suite  of  capabilities  is  typically  comprised  of  modeling  and  simulation;  measurement, 
integration,  hardware-in-the-loop,  and  installed  systems  test  facilities;  and  open-air  ranges.  Both 
the  Department  of  Defense  (DoD)  and  the  United  States  Congress  have  demonstrated  continuous 
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interest  in  the  adequacy  of  Test  and  Evaluation  in  the  acquisition  process.  These  organizations 
have  commissioned  a  number  of  studies  in  an  attempt  to  improve  the  development  processes. 
Many  of  the  recommendations  of  these  studies  have  been  codified  in  law  or  DoD  directives. 

One  recurring  theme  in  these  studies  is  a  clear  correlation  between  system  failure  and  inadequate 
testing  or  an  inadequate  response  to  test  results.11111 

Inadequate  early  developmental  testing  can  be  the  result  of  a  cost  tradeoff  between  high 
and  low  fidelity  testing  and  often  results  in  driving  risk  toward  the  end  of  the  program  where  the 
cost  to  fix  deficiencies  is  high.  Inadequate  testing  is  usually  not  the  fault  of  the  program  office. 
On  many  occasions  the  issue  is  test  facility  availability  or  adequacy.  Test  facility  adequacy  and 
capacity  issues  are  typically  a  result  of  funding  shortfalls,  rigid  or  inadequate  funding  processes, 
inadequate  coordination  with  technology  demonstrations,  or  the  result  of  ineffective  workload 
forecasting. 

Changes  in  the  funding  process  have  made  it  more  difficult  to  maintain  facility  readiness 
and  a  stable  expert  workforce.  The  decision  to  make  funding  available  for  facility 
modernization,  new  capability,  or  new  capacity  is  usually  after  a  major  program  “Milestone  B” 
decision,  well  after  the  facility  operators  need  to  modernize  or  increase  capacity  to  support 
technology  demonstrations,  assessments,  or  early  weapons  system  development.  The  result  is 
that  emerging  technology  projects  and  innovative  weapons  programs  must  rely  on  the 
development  capability  available  to  programs  at  that  time. 

The  collaboration  between  technology  developers,  system  developers,  and  testers  is 
critical  to  ensuring  that  test  facilities  have  the  capability  and  capacity  to  support  all  phases  of  the 
acquisition  system  life  cycle.  When  technologists,  testers,  and  systems  developers  collaborate, 
the  technology  risk  to  programs  is  reduced.  Facility  owners  need  time  to  develop  new  test 
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capability  and  capacity  to  support  programs  effectively.  Changes  in  funding  and  use  policy, 
along  with  improved  workload  and  technology  collaboration,  would  change  the  dynamic  and 
reduce  overall  program  risk  and  cost. 

The  Importance  of  Early  Developmental  Testing. 

Early  in  the  development  of  systems,  testing  is  accomplished  to  obtain  design 
information,  find  defects,  and  reduce  risk  throughout  the  development  process. 1V  In  light  of  the 
critical  technology  demonstrations  and  design  information  needed  by  program  offices,  the  Test 
Community  should  be  seen  by  program  managers  as  a  valuable  part  of  the  program  office  team. 
However,  that  is  often  not  the  case.  A  Defense  Science  Board  (DSB)  task  force  on  Test  and 
Evaluation  Capabilities  stated  that  “A  consistent  theme  the  Task  Force  encountered  during  the 
study  is  that  testing  is  just  another  hurdle  to  be  overcome  in  driving  a  program  past  its  next 
milestone.  The  acquisition  community  views  long  periods  of  testing  as  evidence  of  system 
ineffectiveness  while  testing  itself  is  viewed  as  an  impediment  to  the  systems  success. ”v  New 
fledgling  programs  often  have  close  ties  to  the  technology  programs  that  have  produced  the 
enabling  technologies  needed  by  that  program.  The  use  of  such  high  fidelity  national  test 
facilities  drives  the  risk  of  technology  readiness  down  for  acquisition  programs. 

Cost  and  Quality  of  Testing. 

One  of  the  most  used  excuses  to  abbreviate  a  test  program  is  the  high  cost  of  testing. 
However,  the  effectiveness  of  this  decision  is  not  supported  according  to  a  number  of  defense 
studies. V1VU  One  DSB  Study  finding  was  that  the  cost  of  designing  out  system  or  technology 
shortfalls  is  far  less  than  the  cost  of  fixing  them  later: 

“The  cost  of  testing  is  historically  about  3  to  4%  of  the  total  program  cost.  That  is 
relatively  insignificant.  With  vital  issues  at  stake,  the  minimal  cost  and  very  great 
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value  (return  on  investment)  suggests  we  should  maximize  testing  to  discover  any 
weakness  or  flaws  as  early  as  possible.  Combat  is  the  ultimate  test,  finding  a  fault 
in  combat  is  the  ultimate  cost  of  not  testing .”vm 

The  development  program  manager  sees  test  as  a  large  part  of  his  development  budget. 
But  because  the  sustainment  cost  of  the  program  is  so  high,  the  test  cost  of  the  program  usually 
falls  below  five  percent  of  the  total  program  cost.  Sustainment  costs  are  heavily  dependent  on 
test  adequacy  in  the  development  phase.  A  more  holistic  view  of  test  cost  is  needed.  While  the 
development  phase  program  manager  does  not  directly  manage  the  costs  in  the  sustainment 
phase,  he  is  responsible  for  overall  estimated  program  cost.  The  total  program  cost  must  be 
considered  by  the  system  program  office  throughout  the  program  life  cycle. 

Most  developmental  ground  testing  is  done  in  either  the  system  developer’s  test  facilities 
or  in  national  scale  test  facilities.  In  the  aerospace  sector,  most  national  scale  development  test 
capabilities  reside  within  the  Air  Force,  at  Arnold  Engineering  Development  Complex  (AEDC), 
and  in  the  National  Aeronautics  and  Space  Administration  (NASA)  centers.  Systems  developers 
often  use  lower  cost,  inferior  fidelity  testing  in  lieu  of  national  capabilities  to  reduce  cost  and 
maximize  the  use  of  organic  capabilities.  The  unintended  result  of  the  utilization  of  inferior  test 
facilities  is  that  some  design  flaws  are  not  found.  Program  offices  then  underestimate  the  risk  of 
the  design. 

In  the  late  1990s  all  of  the  services  reduced  funding  for  organic  test  capabilities.  The  Air 
Force  went  a  step  further  and  moved  the  responsibility  for  developmental  testing  to  the  system 
developer. 1X  These  decisions  reduced  test  facility  owner  insight  into  programs  and  made  the 
process  of  collaborating  with  system  developers  more  difficult. 
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The  services  are  often  their  own  worst  enemy  when  it  comes  to  program  oversight. 
Because  of  the  large  size,  limited  number  of  programs,  and  the  difficulty  getting  new  programs 
approved,  the  services  are  reluctant  to  delay  or  cancel  programs.  Performance  problems  in 
system  development  are  often  the  result  of  broad  waivers  to  testing  granted  to  programs  by  the 
parent  services  and  the  dismissal  of  test  results  by  the  programs  themselves. x  A  2008  DSB  Task 
Force  Report  on  Developmental  Test  and  Evaluation  highlighted  the  issue  of  programs  having 
increasingly  high  Initial  Operational  Test  and  Evaluation  (IOT&E)  suitability  failure  rates. X1 
“Early  in  the  DSB  study  it  became  obvious  that  high  suitability  failure  rates  were  a  result  of 
systematic  changes  that  had  been  made  to  the  acquisition  process;  and  that  changes  in 
developmental  test  and  evaluation  could  not  remedy  poor  program  formulation. ”xu  At  the  very 
beginning  of  a  program’s  life  cycle  the  program  is  primarily  constrained  by  cost  and  contracting 
issues.  By  passing  along  the  cost  of  testing  to  programs,  funding  policy  incentivizes  program 
managers  to  minimize  testing  in  early  development.  Early  development  is  precisely  when 
acquisition  programs  need  the  most  information  out  of  early  development  test  facilities. 

The  cost  of  developmental  ground  testing  is  broken  down  into  both  direct  costs,  the  costs 
attributed  to  executing  the  test  project,  and  the  indirect  costs,  the  costs  associated  with  facility 
and  base  ownership.  The  direct  cost  of  development  ground  testing  can  be  broken  down  to  two 
basic  costs.  These  costs  are  manpower  and  consumables.  The  consumables  are  dominated  by 
fuel  and  power  costs.  Since  manpower  must  be  managed  over  the  long  term,  funding  of 
manpower  as  an  indirect  cost  would  alleviate  problems  with  maintaining  expertise.  The  indirect 
costs  provide  for  security,  base  infrastructure,  facility  upgrades,  routine  and  scheduled 
maintenance,  and  other  overhead  costs. 
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Decisions  about  who  will  utilize  a  facility  or  how  the  cost  of  that  use  is  funded  should  be 
made  at  the  level  that  best  supports  Air  Force  interests.  “Improving  the  value  of  testing,  not 
reducing  the  cost  of  testing,  should  be  the  goal  in  all  future  decisions  regarding  consolidation  of 
activities,  investment  planning,  and  test  resource  management  throughout  the  Department  of 
Defense. ”xm  Program  risk  is  ultimately  taken  by  the  service  even  when  fixed  price  contracts  are 
used  for  the  acquisition.  Since  the  department  assumes  the  risk,  decisions  about  an  individual 
program’s  test  cost  may  not  be  in  best  interest  of  the  department.  Facility  readiness,  capacity,  or 
another  program’s  needs  might  outweigh  an  individual  program’s  interests.  Funding  and  use 
policy  could  alleviate  this  issue  by  moving  the  decision  process  from  individual  programs  to 
stakeholders  in  the  community  at  large. 

Funding  and  Use  of  National  Test  Capabilities. 

After  the  inception  of  the  AEDC  in  1951,  testing  was  conducted  as  a  cost- free  service  to 
programs.  Institutional  funding  for  testing  was  provided  by  the  service.  This  funding  was 
utilized  to  perform  test  and  evaluation,  maintenance  of  test  capabilities,  test  technology 
development,  new  capability  planning,  base  support  and  overhead  functions.  Programs 
competed  for  access  to  these  “best  in  the  world”  capabilities  based  on  program  priority.  During 
these  years,  one  would  think  that,  lower  priority  programs  and  technology  development 
programs  would  not  be  unable  to  get  access  to  best  in  class  test  capabilities.  However,  due  to 
weapon  development  program  scheduling  constraints,  more  than  30%  of  AEDC’s  capacity  went 
to  supporting  system  technology  development  efforts. X1V  In  the  early  1970s  the  institutional 
funding  accounts  were  abolished  in  favor  of  program  funded  testing.  Facility  workload 
collapsed  as  program  offices  and  system  developers  eliminated  tests  from  their  programs,  found 
lower-cost,  inferior  capabilities  to  utilize,  or  constructed  their  own  capabilities  at  the  prime 
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contractor’s  location.  After  the  near  closure  of  AEDC,  Air  Force  funding  became  a  mix  of 
institutional  and  program  funding.  Today  the  direct  cost  of  testing,  operations,  and  non-routine 
maintenance  is  borne  by  the  programs  utilizing  the  capability.  Overhead  functions  such  as 
sustainment,  base  support,  and  modernization  are  funded  by  institutional  accounts.  Today 
almost  none  of  the  test  capacity  in  Air  Force  development  test  capabilities  is  currently  being 
utilized  by  technology  developers.  The  result  is  that  technology  programs  must  utilize  lower- 
cost,  inferior  test  capabilities  while  capacity  at  the  best  facilities  goes  unused. 

In  2002  NASA  implemented  this  same  funding  model.  The  outcome  was  predictable.  A 
number  of  facilities  closed  and  workload  dried  up.  The  DoD  commissioned  studies  by  RANDXV 
and  the  Institute  for  Defense  Analysis  (IDA)XV1  to  understand  the  impact  of  the  closures  and  the 
funding  policy  changes  on  DoD  programs.  The  resulting  interest  by  the  Office  of  the  Secretary 
of  Defense  (OSD)  and  Congress  resulted  in  an  increase  in  the  NASA  Aeronautics  budget,  which 
slowed  the  closure  of  NASA  facilities.  Recently  NASA  has  returned  to  a  policy  of  not  charging 
for  the  use  of  test  facilities  when  they  deem  it  is  in  their  interests.  The  DoD  will  benefit  where 
interests  overlap.  The  downside  to  this  change  is  that  DoD  program  will  move  to  NASA 
facilities  when  they  can  get  free  testing  reducing  the  collaboration  between  DoD  testers  and  the 
DoD  technology  and  system  development  organizations.  This  reduced  collaboration  between 
DoD  testers  and  DoD  technology  developers  hinders  development  of  organic  engineering  talent 
within  the  DoD.  This  talent  is  needed  to  support  the  system  development  phase  of  the  program. 

Technology  Collaboration 

One  way  for  development  testers  to  forecast  new  needs  and  prepare  for  the 
weaponization  of  new  technologies  is  to  collaborate  on  the  maturing  of  these  weapons 
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technologies.  This  approach  is  hampered  by  the  developmental  test  charge  policies  and  the  lack 
of  funding  available  to  use  for  such  collaborations. 

The  gulf  in  technology  transfer  between  the  laboratory  and  acquisition,  sometimes  called 
the  “valley  of  death”  by  technologists,  has  grown.  In  addition  to  funding  constraints, 
collaboration  between  DoD  technology  developers  and  testers  is  hampered  by  organizational 
constraints  and  a  lack  of  a  collaboration  policy.  Technologists,  who  had  access  to  the  very  best 
test  capabilities  before  1970,  no  longer  have  a  low-cost,  high-fidelity  way  to  demonstrate 
capabilities.  Technologists  and  acquisition  programs  no  longer  have  a  collaborator  relationship; 
instead,  the  relationship  became  transactional. 

Testers,  technologists,  and  program  managers  can  all  benefit  from  a  relationship  based  on 
collaboration.  Technologists  benefit  by  developing  and  demonstrating  their  technology  in  the 
facilities  with  the  best  flight  simulation  fidelity  and  the  lowest  measurement  uncertainties 
available.  They  gain  insight  by  having  professional  experimentalists  develop  test  plans  that  will 
provide  the  very  best  insight  into  the  technology’s  maturity.  These  benefits  have  narrowed  the 
“valley  of  death”  for  technology  development. 

Testers  gain  a  great  deal  of  insight  through  collaboration  with  technologists.  They  can 
explore  new  test  techniques  and  capabilities  to  improve  simulation  quality  and  reduce 
measurement  uncertainty.  It  allows  them  to  better  understand  what  will  be  needed  to  transition 
this  new  technology  to  weapons  programs.  Testers  develop  insight  into  where  technology  is 
headed  and  where  investment  is  needed  to  improve  test  capability,  capacity,  and  technology  to 
support  the  future  Air  Force. 

Program  managers  benefit  from  insight  into  the  weapon  system  gained  by  in-depth 
familiarity  with  the  technologies  that  the  program  managers  seek  to  transition.  They  gain 
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improved  test  quality  and  efficiency  from  new  test  techniques  and  test  capability  improvements 
developed  during  the  technology  development  phase. 

The  Air  Force  gains  a  workforce,  infrastructure,  and  process  that  has  the  opportunity  to 
provide  the  best  possible  weapons  at  the  lowest  possible  cost  and  shortest  possible  schedule.  All 
three  communities  gain  a  developed  workforce  that  has  better  insight  and  understanding  of  the 
entire  acquisition  process  and  the  technologies  that  are  being  matured. 

When  testers,  technologists  and  program  managers  compete  for  funds  and  each 
organization  is  trying  to  protect  their  funding  lines  at  all  cost,  little  collaboration  results.  For  this 
important  collaboration  to  succeed,  each  community  must  be  funded  to  collaborate. 
Developmental  testers,  technologists  and  acquisition  program  managers  must  become  allies  not 
competitors.  The  resulting  advanced  concept  demonstration  programs  could  result  in  more 
technologies  successfully  transitioned  with  shorter  acquisition  schedules. 

Capacity  and  Facility  Planning 

Public  Law  81-415:  Unitary  Wind  Tunnel  Plan  Act  of  1949  and  the  Air  Engineering 
Center  Act  of  1949  established  AEDC  and  several  major  NASA  test  capabilities.  Congress 
showed  support  for  General  Henry  Arnold’s  vision  that  the  United  States  never  be  caught  behind 
in  aerospace  technology. xvn  Congress  directed  that  the  administrator  of  NASA,  then  called 
National  Advisory  Committee  for  Aeronautics  (NACA),  and  the  Secretary  of  Defense  develop  a 
unitary  plan,  a  coordinated  plan  to  develop  wind  tunnels  in  DoD  and  NACA,  and  provide 
updates  on  that  plan  to  Congress  in  accordance  with  scientific  advances.  These  updates  have 
never  been  provided  and  thus  this  method  to  secure  funding  for  new  capability  has  never  been 
utilized  by  the  Air  Force. 
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Test  facility  owners  are  challenged  to  provide  adequate  capacity  and  capability  to  support 
early  development.  These  limitations  stem  from  limited  insight  into  the  test  capability  needs  for 
new  systems  in  a  timeframe  that  supports  development  of  new  test  capabilities  first,  a  budgeting 
preprocess  that  does  not  support  development  of  new  test  capability  of  capacity  until  there  is  a 
“Milestone  B”  program,  a  funding  process  that  competes  early  development  test  priorities 
directly  against  flight  and  IOT&E  testing  needs,  and  a  lack  of  policy  to  support  tester 
involvement  in  technology  development. 

Due  to  the  funding  prioritization  process,  funding  for  capacity  and  modernization  is 
usually  not  available  in  time  to  modernize  the  capability  for  innovative  systems.  Air  Force 
modernization  funding  is  planned  and  programmed  using  the  Test  Investment  Planning  and 
Programming  (TIPP)  Process. xvm  This  program  competes  modernization  needs  across  the  Air 
Force  test  enterprise.  For  a  need  to  be  considered,  there  must  be  a  validated  test  need  to  support 
a  “Milestone  B,”  or  later,  acquisition  program.  The  TIPP  program  funding  is  allocated  across  the 
Five  Year  Development  Plan  (FYDP).  This  process  results  in  new  test  capability  needs  not  even 
being  considered  in  the  process  until  five  to  ten  years  after  the  capability  is  needed.  The  result  is 
that  TIPP  investment  in  early  development  test  capabilities  is  limited  to  large  maintenance  or 
recapitalization  projects. 

OSD  investment  in  test  modernization  is  accomplished  through  the  Central  Test  and 
Evaluation  Investment  Program  (CTEIP).X1X  Since  the  services  have  been  gatekeepers  to  the 
CTEIP  process,  early  development  test  needs  are  not  submitted  or  are  really  service  investment 
and  modernization  projects  rather  than  new  test  capabilities. 

Only  two  new  major  test  capabilities  have  been  constructed  at  AEDC  since  the  1970s. 

The  first  was  the  Aeropropulsion  Systems  Test  Facility  (ASTF)  complex  and  the  second  was  an 
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altitude  solid  rocket  motor  test  facility  called  J6.xx  The  first  was  the  result  of  a  herculean  effort 
that  took  support  from  the  White  House,  Congress,  and  OSD.  The  DoD  allocated  the  entire  DoD 
Military  Construction  Budget  (MILCON)  budget  for  one  year  to  fund  the  project.™  Without  the 
ASTF  test  capabilities  the  DoD  could  not  have  developed  the  F22  and  F35.  The  J6  project  was 
the  result  of  a  catastrophic  rocket  engine  detonation  that  destroyed  the  J5  test  facility  in  the 
middle  of  a  strategic  weapon  acquisition  system  development.™1  Both  of  these  projects  were 
funding  emergencies  that  were  spawned  out  of  crisis.  Without  the  emergencies,  neither  would 
exist  and  it  is  unlikely  that  just  the  right  emergency  will  occur  to  ensure  the  Air  Force  creates  the 
development  capabilities  it  needs. 

The  largest  factor  in  development  capacity  is  funding  to  support  the  workforce  and 
facility  maintenance.  Budget  cuts  between  1991  and  2015XX111XX1V  left  the  DoD’s  test  and 
evaluation  workforce  decimated.  Workload  is  increasing  at  an  all-time  high  pace.  Recently 
funding  for  the  reactivation  of  test  units  to  support  the  fast  growing  workload  has  added  to 
budget  pressures.  Funding  has  started  to  increase,  but  the  funding  and  the  time  needed  to 
acquire,  train,  and  develop  test  experts  is  just  not  able  to  keep  pace  with  the  quickly  increasing 
test  pace.  An  additional  impediment  to  adequate  early  development  test  funding  is  the 
prioritization  process.  Air  Force  Material  Command  now  uses  the  Capability  Analysis  and  Risk 
Assessment  (CARA)XXV  process  to  prioritize  efforts  for  the  execution  and  budget  years. 

Operating  instructions XXV1  across  the  Air  Force  Material  Command  (AFMC)  are  being  updated  to 
make  this  process  standard  across  the  test  enterprise.  This  process  has  two  major  shortfalls  as  it 
pertains  to  capacity.  The  process  competes  early  development  needs  against  higher  priority  more 
mature  development  needs  and  it  is  not  forward-looking  enough  to  support  large  increases  in 
workload. 
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Recent  Progress 


One  recent  example  of  where  technologists  and  testers  have  collaborated  is  in  Supersonic 
Combustion  Ramjet  (SCRAMJET)  research.  The  results  of  this  collaboration  were  mixed.  The 
OSD  Test  Resource  Management  Center  (TRMC)  funded  Test  and  Evaluation  Science  and 
Technology  (T&E  S&T)  program  provided  funds  to  AEDC  to  develop  test  technology  for  use  in 
test  conduct  and  analysis  of  high-speed  weapons.  The  Air  Force  Research  Laboratory  (AFRL) 
and  AEDC  collaborated  together  to  advance  both  system  and  test  technologies.  The  program 
resulted  in  state-of-the-art  award  winning  laser  diagnostics,  a  world  unique  vitiation  affects 
database,  world  unique  test  facility  heater  technology,  thrust  accounting  systems,  computational 
models,  performance  models,  analysis  techniques,  innovative  flight  test  techniques  and 
capabilities,  and  a  competitive  engine  program  for  cruise  missile  development.  The  most 
important  aspect  of  this  collaboration  is  the  advancement  and  development  of  the  workforce  in 
AFRL,  AEDC,  and  the  system  developers.  Everyone  was  able  to  leverage  funding  from  others 
to  make  more  progress  than  they  could  have  possibly  made  alone.  However,  this  collaboration 
fell  apart  when  funding  competition  became  the  issue.  The  Defense  Advanced  Research 
Program  Agency  (DARPA)  took  over  as  the  lead  agency  for  the  program. xxvu  NASA  offered 
free  support  and  testing  so  DARPA  decided  to  collaborate  with  NASA  and  locked  out  both 
AEDC  and  AFRL.  No  one  at  AEDC  is  involved  in  the  program,  and  only  a  few  people  from 
AFRL  are  involved.  The  facilities  utilized  are  not  capable  of  supporting  the  follow-on  weapons 
acquisition.  When  the  program  attempts  to  transition  from  DARPA  to  an  acquisition  program, 
there  will  be  few  experts  in  the  Air  Force  who  can  support  the  development  or  testing  of  the 
system.  It  will  take  considerable  time  and  cost  to  develop  the  workforce  and  capabilities  to 
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support  the  Air  Force  acquisition.  The  result  will  be  long  delays  in  the  acquisition  of  a  weapon 
system  based  on  these  technologies. 

Recent  increases  in  workload  have  helped  bring  attention  to  the  funding  shortfalls  in 
early  development  test.  This  recent  increase  in  workload  can  be  attributed  to  two  things.  The 
first  is  a  reinvigoration  of  the  test  rigor.  This  reinvigoration  of  test  rigor  provides  real  insight  to 
program  managers  and  system  developers,  as  opposed  to  just  providing  mountains  of  data.  The 
second  is  a  focus  on  owning  the  technical  baseline  directly  with  program  offices.  Program 
offices  have  begun  to  fund  testing  directly,  instead  of  through  prime  contractors.  Test  analysts 
have  become  invaluable  to  program  offices.  Programs  have  requested  support  developing  test 
plans,  analyzing  data,  developing  Requests  for  Proposals  (RFPs),  developing  Requests  for 
Information  (RFI),  and  evaluating  proposals.  The  need  for  analysts  to  support  internal  testing, 
customer  testing  at  other  locations,  and  program  office  support  has  led  to  a  severe  shortage  of 
personnel.  Internal  policies  for  hiring  both  organic  test,  analysis,  and  support  contractor 
workforce  have  limited  AEDC’s  ability  to  support  internal  and  external  customers.  Another 
factor  limiting  support  to  the  increased  workload  is  the  availability  of  test  units.  Because  there 
was  little  or  no  interaction  with  the  programs  in  the  technology  demonstration  phase,  the 
increased  workload  is  not  projected.  When  the  technology  phase  workload  is  done  outside  of 
DoD,  not  only  does  the  valuable  collaboration  not  occur,  but  often  this  lack  of  workload  can 
result  in  the  closure  of  test  capability  that  could  be  needed  in  the  future. 

An  example  of  where  funding  is  needed  to  create  increase  capacity  is  in  the  Propulsion 
Wind  Tunnel  Complex  at  AEDC.  In  the  sixteen-foot  transonic  wind  tunnel  (16T),  a  new  test  and 
model  support  cart  is  needed  to  provide  enough  capacity  to  support  the  upcoming  workload. 

This  cart  is  40-foot  x  40-foot  x  40-foot  section  of  facility  that  is  removed  with  the  test  article  and 
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weighs  nearly  one  million  pounds.  The  cost  of  the  cart  is  approximately  $30M  and  could  take  as 
long  as  two  years  to  construct.  In  fiscal  year  2018,  this  cart  would  save  nearly  18  weeks  of 
schedule  in  16T  at  a  value  of  nearly  $4.5B  in  program  delays.  A  similar  savings  is  projected  in 
2020.  It  is  unlikely  that  the  funding  process  will  result  in  the  project  getting  the  needed  priority 
so  that  funding  is  available  to  purchase  the  cart  in  time  to  support  testing  in  2020 

The  Air  Force  developmental  test  community  is  trying  to  implement  OSD  policyxxvm  on 
early  tester  involvement  recommended  in  the  2008  DSB  report  on  Developmental  Test  and 
Evaluation. XX1X  This  policy  requires  the  determination  of  a  Responsible  Test  Organization  (RTO) 
early  in  the  program.  One  failure  of  the  policy  is  that  the  acquisition  program  manager  can 
decide  if  this  level  of  scrutiny  is  really  needed,  and  they  can  limit  support  by  limiting  funding  to 
the  RTO. 

The  collaboration  between  the  OSD  T&E  S&T  program  yielded  the  technologies  to 
construct  a  suite  of  new  test  capabilities  to  support  developmental  testing  of  new  hypersonic 
weapons.  While  the  services  did  not  submit  those  test  capabilities  as  needs  to  CTEIP,  OSD 
TRMC  (the  organization  that  executes  the  T&E  S&T  and  CTEIP)  secured  $350M  in  the  FY 17 
budget  to  develop  these  capabilities.  The  gains  made  by  this  program  could  be  expanded  to 
cover  other  flight  regimes. 

Program  competitions  could  be  evaluated  on  wind  tunnel  based  performance  estimates. 
Acquisition  demonstration  program  fly-offs  are  very  costly  and  can  often  have  limited  value  to 
the  system  development  process.  These  unique  systems  are  designed  around  the  flight 
demonstration’s  grading  rubric  not  on  the  system’s  mission  requirements.  After  the  demo  is 
complete,  a  redesign  of  the  system  is  accomplished  to  meet  the  system  mission  requirements. 
This  leads  the  flight  demonstration  to  being  more  stunt  than  system  demonstration.  At  least  one 
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program  is  currently  conducting  the  “fly-off’  in  the  wind  tunnel  where  force-accounting  inputs 
and  stability  derivative  inputs  can  be  acquired  with  common  test  techniques  and  low 
uncertainties.  This  data  can  then  analyzed  against  the  mission  requirements.  The  system 
performance  of  competitors  can  be  compared  in  a  direct  and  precise  way  avoiding  protests  on 
performance  evaluation  criteria.  This  competition  should  be  monitored  and  the  technique  should 
be  evaluated  for  expansion  to  other  programs. 

Recommendations. 

The  test  community  needs  the  ability  to  develop  and  maintain  a  stable  workforce. 
Conflicts  between  internal  priorities,  external  customer  priorities,  and  limitations  on  funding  type 
create  inefficiencies  in  project  execution  and  workforce  development.  AEDC  needs  the  ability  to 
shift  funding  and  internal  project  execution  to  enable  the  most  efficient  execution  of  the  test 
mission.  This  could  be  accomplished  in  a  number  of  ways  including  institutional  funding 
schemes  or  a  single  program  element  funding  strategy.  While  single  program  element  funding 
would  add  flexibility,  it  could  add  risk  to  the  budget  by  making  it  a  target  of  detractors.  An 
additional  benefit  to  the  institutional  funding  model  is  the  possibility  of  high  utilization  of  test 
capabilities  by  technology  developers.  Technology  developers  could  have  more  demonstrated 
technologies  ready,  on-the-shelf,  to  support  acquisition  programs. 

The  RTO  needs  to  be  independent  and  the  test  team  needs  to  predate  the  “Milestone  B” 
decision  for  the  program  itself.  For  the  RTO  concept  to  be  effective,  funding  decisions  for  the 
RTO  team  need  to  be  made  by  the  test  community,  not  by  the  program  office. 

The  Air  Force  needs  a  valid  process  that  can  lead  to  new  test  capability  to  support  the 
development  of  innovative  weapon  systems.  Today  no  real  funding  process  exists  in  the  Air 
Force.  Successes  by  OSD  in  this  regard  need  to  be  expanded  outside  of  hypersonic. 
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Collaboration  is  the  key  to  developing  a  better  test  capability  base.  This  collaboration 
will  improve  workload  forecasting  and  develop  a  better  workforce.  The  collaboration  between 
testers,  technologists,  and  program  offices  enabled  by  the  OSD  T&E  S&T  program  needs  to  be 
expanded  to  support  development  in  other  flight  regimes. 

The  cost  of  testing  is  borne  in  the  acquisition  community  it  supports.  To  provide  for  a 
higher  fidelity  and  more  robust  test  capability  base,  funding  for  testing,  excluding  consumables, 
should  be  institutionally  funded  by  the  test  community.  This  funding  policy  would  result  in 
programs  being  incentivized  to  conduct  the  proper  testing  and  testers  being  able  to  provide  the 
right  test  capability  and  capacity  mix. 

Ground  test  “fly-off’  techniques  should  be  considered  as  a  tool  in  evaluating  competing 
system  proposals.  This  technique  could  save  considerable  time  and  cost  in  comparison  with 
traditional  fly-off  demonstrations. 

Conclusions 

During  the  dedication  of  AEDC,  President  Truman  talked  about  the  need  to  maintain  the 
very  best  Air  Force  and  AEDC’s  role  in  maintaining  that  preeminence. xxx  The  best  systems  are 
the  result  of  the  best  engineers  utilizing  the  very  best  tools.  Developmental  test  is  one  of  these 
tools.  Arguably  testing  has  one  of  the  largest  impacts  on  system  suitability  when  compared  to 
other  systems  engineering  tools.  It  is  important  to  encourage  the  use  of  these  best  capabilities 
and  maintain  expertise  through  their  use.  It  is  also  important  to  ensure  facilities  are  up  to  date 
and  have  the  ability  to  support  both  test  technology  and  systems  development  programs.  The  Air 
Force  funding  and  use  policies,  as  they  pertain  to  early  developmental  testing,  should  be  codified 
in  a  way  to  support  these  goals.  To  accomplish  these  goals  a  number  of  funding  and  use  policy 
changes  should  be  investigated.  These  changes  include: 
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•  flexible  funding  options  are  needed  to  enable  the  shifting  of  funding  to  support 
workload  and  priority  changes; 

•  RTO  funding  needs  to  be  independent  of  program  office  funding; 

•  a  funding  process  needs  to  be  developed  that  supports  the  development  of 
innovative  new  test  capabilities  to  support  new  acquisition  programs; 

•  a  process  is  needed  to  provide  funding  for  testers  to  collaborate  with  technology 
developers; 

•  the  OSD  T&E  S&T  program  should  be  expanded  to  additional  flight  regimes; 

•  the  wind  tunnel  “fly  off  program  should  be  closely  monitored  and  if  successful 
expanded  to  early  system  and  technology  development  programs; 

•  the  cost  of  testing,  except  for  consumables,  should  be  institutionally  funded. 

These  changes  would  not  increase  the  total  cost  of  testing  inside  the  Department.  These 

changes  would  result  in  lower  cost  and  faster  development  of  aerospace  systems  with  much 
improved  performance.  A  lower-cost,  faster-development  cycle  could  yield  savings  that  could 
be  invested  in  new  technologies  and  new  weapon  systems.  This  faster  development  pace  could 
help  the  U.S.  maintain  its  advantage  in  the  air. 
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